Part 1: Lightweight Object Detection¶

Loss Curves¶

5 Sample Detections¶

Personal Images¶

In summary, my SSD-lite banana detector successfully learns to localize a single “banana” class on the D2L dataset. The training loss curves show clear convergence, and on both validation images and my own banana photos, the model usually produces reasonable, although slightly loose, bounding boxes around the fruit. When the scene looks similar to the training data (a single, reasonably large banana on a clean background), the detector typically draws one box that covers most of the banana, which shows that the backbone, anchors, and loss design are sufficient for this simple setting.

However, I also observe a number of clear failure cases that highlight the limitations of my approach. When the banana is very small in the image or placed near the edges, the detector often either misses it entirely or fires on a nearby background region instead. In more cluttered scenes (e.g., banana on a messy desk or next to other yellow objects), the model sometimes places boxes on textures or colors that resemble the banana rather than on the banana itself, suggesting that the learned features are relatively shallow and sensitive to color rather than higher-level shape. On some of my custom images, where the banana is partially occluded or at an unusual orientation, the predicted box can drift and only cover part of the fruit, or become oversized and include a large chunk of background.

These behaviors are consistent with the design choices I made: a lightweight SSD-lite architecture with a tiny backbone, a single low-resolution feature map, coarse anchors, and training from scratch on a small, single-class dataset. A heavier detector such as Faster R-CNN with a pre-trained backbone and multi-scale features would generally handle small objects, clutter, and pose variation more robustly, but at a higher computational cost. In that sense, the inaccuracies I observe are not just random errors. They are directly tied to the architectural trade-offs I made to keep the model simple and efficient.

Part 2: Non-Maximum Suppression (NMS)¶

Discussion of Results (Images in Part 1)¶

For this project I implemented a standard greedy NMS routine that sorts boxes by score, repeatedly keeps the highest-scoring box, and removes all remaining boxes whose IoU with it exceeds a chosen threshold. I then compared it directly with torchvision.ops.nms by running both on the same decoded boxes and scores from my SSD-lite detector and visualizing the results. In practice, the two methods produced almost identical outputs on all test and personal images: the same 1–3 boxes were kept after suppression, and any differences were minor (usually just which of two nearly overlapping, similar-score boxes survived). This confirms that my implementation matches the behavior of the PyTorch reference.

Conceptually, NMS’s purpose is to clean up the dense set of overlapping anchor predictions and turn them into a small set of final detections (ideally one box per object). It does not make the model more accurate by itself. It only removes redundant boxes. The main limitations I observed are:

(1) NMS blindly trusts the model’s scores, so if the detector assigns higher scores to a wrong region (e.g., my hand or the cabinet instead of the banana), NMS will keep that wrong box.

(2) It cannot fix localization errors. If all high-scoring boxes are slightly off, then final box will also be off.

(3) It depends on a hand-chosen IoU threshold, which can either suppress too many boxes or leave duplicates if set poorly.

Part 3: Human–Object Interaction (HOI) Analysis using VLMs¶

For the HOI part, I used GPT as a vision–language model on three images: (1) a mounted police officer on a horse, (2) a motorcycle racer, and (3) a person reading a book. I prompted the model with: “List all human–object interactions in this image using the format (e.g., ride bicycle, hold phone).” GPT correctly produced interactions such as “ride horse / hold reins / wear helmet” for the first image and “read book / hold book” for the third image. These cases are relatively easy: the human and object are clear, and the interaction matches very common categories seen in training. The main failure case came from the motorcycle image. GPT sometimes hesitated between “ride motorcycle” and more generic descriptions like “lean over motorcycle” or “work on motorcycle”, likely because the racing posture and motion blur make the interaction visually ambiguous. I tried prompt refinements such as “Describe the main action the person is performing with the object (e.g., ride motorcycle, not stand near motorcycle).” This reduced but did not completely remove the ambiguity. Overall, GPT performs well on clear, everyday HOIs but can still misclassify actions when poses are extreme or the visual evidence is ambiguous, and prompt engineering only partially fixes these edge cases.